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during implementation or after proiect completion* Numerous tables 
lis* di*rectors* responses to individual items on the survey 
auestionnaire* (CS) 



* - Reproductions supplied, by BDRS are the best that can be made. ' * 

* from the original document* ^ * 
**^*********;it?it** ************************************************ ******* 



ERLC 



BROAD VIEW OF CAUSE PROJECTS: 
SURVEY OF PROJECT DIRECTORS 
AND ANALYSES OF PROPOSALS 



Kathleen Porter 



O-S DEPARTMENT OF HEALTN. 
EDUCATION 4MCLFARC 
NATIONAL INSTITUTE OF 
Et>UCAT»ON 

THIS OOCgAAENT HAS BEEN HCPftO- 
OUCED EXACTLY iS RECEIVED FSOM 
THE PERSON Oft ORGANIZATION OHlGiN- 
ATiSOrT PO_iNTS OF V<EW 0*J OPINIONS 
STAT^O OO NOT f^CESSAHiLY fiEPRE' 
SENT OFFICIAL NATiONAL iNSTrTUTE OF 

eoyCATiON POSITION or policy 



^^'PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDgCATlONAL RESOURCES 
V ^ INFORMATION CENTER (ERIO" 



V55 



• . ' . . BROAD VIEW OF CAUSE PROJECTS 

. Introduction ' ■ ' „ 

Kathleen Porter . ' ' , 

The- CAUSE Program is four years old, A program evaluation has been ' ^ 
conducted by Development & Evaluation Associates with, the aim of refining ; 
and improving the CAUSE program* This evaluation provides a description* and 
analysis o,f the CAUSE program- The specific issues of the evaluation are: 
(1) thie'^xtent to which high priority local college and university needs are 

^ t 

being met bty locaVCAUSE projects; .(2) the ways in which CAUSE projects are 
being implemented; (3) the extent to which instructional improvement is 
resultifig from CAUSE; (4) the nature and use of evaluation data on CAUSE 
projects; (5) the relative costs of, the functional activities olCAUSE pro- 
jects and how they relate tp post-CAUSE. institutional support; and .(6) pro- 
gram changed and modifications to be made in the CAUSE program. 

This paper contains analyses' of data gatheried using the broad focus 
^techniques of the evaluation- -In contrast to the narrow ih-<lepth focus 
provided by the eight case studies of CAUSE projects and the somewhat ^ 
broader focus provided by the. 17 site visits, the lactivities described in 
this paper - the analysis of 273 funded-.CAUSE proposals, and the surveying' 
0f^201 CAUSE project directors - provide a broad view of all the CAUSE 
projects funded between. 1976 and 1978. . ' 

^ The data gathered frm the CAUs/proposals find fromjthe, survey of 
*, \ ' ■ 1 ■ ' . \ ' 

project' directors have been aggregated for the purpose of discovering 

trends, Ajs with most quantitative methods^ these were utilized for the 

pUrposie of Justifying broad statements aboujt CAUSE projects". In contrast* 



f ■ ■ \ \ . 2 

r ■ ■ ■■■■ ' . ' ■■■ ■■ - 

* ' ^ ' ' - ^ - ^ * 

.the-^ualitative methods used in the site visits artd the case studies were 
se^isi^e to peculiarities and individual differences* - 

The broad focus evaluation relied upon a- quantitative content analysis 
\ of CAU^E proposals and the use of programs in SPSS (Statistical Packages 
foi^ the Social Sciences) for analysis of the surveij^ of project directors* 
In contrast, the narrow and. medium focus evaluations. relied'uponNjbserva- 
. tion* > 

The content analysis of CAUSE proposals provides ah overview M)f the 
way projects were described before they were implemented. It portrays .ther 
intended recipients of. the benefits of the proposed projects; it describes 
the needs as they were perceived by the proposers; it displays the goals 
and (Objectives of the proposed projects; it portrays the proposed strategies 
for meeting needs and realizing goals. , - 

The survey of CAUSE project directors, on the other ha*wJ,. portrays 
the projects as seen by project directors either during implementation or 
after project completion* It displays the amount and kinds of jnodifica- 
,tions made in the projects since the proposal was writteji; it portrays 
those activities project directors describe as important fori^assuring 
success; it displays those factors which project directors describe as 
having been a problem for th^r projects; it describes project directors ' 
attitudes toward project ev^iluatton and recommendations they, would make 
to NSF and to prospectiveyproject directors* 

Taken together, the/two broad .focus techniques provide, a Beford and 
after snapshot of the c/uSE program as a whole* 

' / . ' • 
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; SURVEY OF CAUSE PROJECT DIRECTORS, 1976-1978 ^ 

Summary 

This section describes the findings from the Survey of Project 
Directors"^ It begins by describing the purposes of the survey, its place 

in the overall evaluation of the CAUSE program, the development of the two ^ 

> 

versions of the survey, and the procedures used in obtaining and analyzing 
the survey data/ 

Next, general findings are summarized within five of 1;he six Issues 
l^ntiiied as those informing the evaluation as a whole: institutional need* 
project implementation, Instrtictional improvement, evaluation, and recom- 
mendations for modifications* 

Specific findings are compared between those questions Which are 
coninpn^ or at least similar, to, the two* versions of the stirv£y. Tables 
follow which present those comparison^ in terms of percentage of project 
directors from each of the two samples who chose each option to each 
question* ' : . < ^ ' 

At the end of the section are tables summarizing project directors^ 
responses, first to Survey One and then to Survey Two, These tables pre- 
sent the percentage of project directors who chose each -option to each 
item to both forced-choice and open-ended questions, * , 
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Development and Procedures of the Survey 

This survey represents one of the broad focus data collection tech- 
^ piques of the evaluation of the National Science Foundation's Comprehensive 
* .Assistance to Undergraduate Science Education (CAUSE) program, The* broad 
focus evaluation techniques, which include both the survey and the content 
, analysis of funded proposals, prov^ide an overview of the total CAUSE 
program in ddntrast to the site visits which provide a medium range view 
and ti)e case- stucJfes which provide a close-up portrayal of some sample 
projects, ' 
^ The survey provides reliable information on program-wide issues by ' 
aggregating the responses of virtually all the pr'oject directors on cri- 
" tical aspects of their projects like project ijnplementation, impact, 
evaluation, and overall project success in the case of the first random 
Sample of project directors, and project characteristics, implementation, 
impact, evaluation, and reconmendations in the casfi of the second random 
sample of project directors » 

Purpose 

The function of the survey is to extend and broaden the primary data 
♦ 

bases to inc>«de^the critical insights of project directors. No other 
CAUSE staff participated.. It is our belief that project directors have a 
.' unique perspective and can provide important ififormatj^n about the 

r- • 

functioning of CAUSE projects at the local level. The purposes of the 
surv^ are to identify the diversity of views held hy;project directors on 
:tbe critical evaluation issues and to confirm ^or disconTirm observations 
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made during the 17 site' visits and over the course Of the eight case studies 

The' specific goal of the survey is to focus on CAUSE projects 
in.theway.in which they evolve. The topics covered by the survey inclutfe 
how CAUSE projects get implemented, what kind of impact projects have on, - 
inSfi^tutions, how decisions are made as the project progresses, and how 
successfjfl projects are. These issues h^ve. been found to.be import^nj^o 
the success of .instructional improvement projects accor'ding to literature 
on instructional development in. higher education (e.g, , Diamond et al., 
' 1975;'Durzo^ 1976; Holsclow, 1974). 

Surveys, in general ^ have some very definite strengths and weaknesses. 
The major advantage is the ability to gather data from a representative ■ 
sample by methods which are more efficient than those of direct inter- 
actions with or observation of project staff: In the. case of this * 
. evaluation. tbfiJ^epres&ntativey«i^of the sample is assured since the two- 
stage survey process surveys/all funded C^USE projects, 19^76-78.; 
Bias in sampling is one limitation of surveys which this evaluation avoids. 

However, another limitation' of surveys-*that ir,'the superficial l&vel to 



which issues may be studied— is important to consider, using surveymg 
as only one met^hod in a multiple method evaluation this study has built - 
into it (JpportunH^iesr to study all issues in depth with a. small number 6f 

cases- The weakness of the survey is overcome by utilization' of 

^ / . ^ ^ -\ , ' ^ ■ V ^ ' ^ 

other methods which permit mqre irr-depth analysis* ■ . » - : 

?thod of ,the Survey ' - 

/ . ' ' ■ \ ■ ' ' ' ^ 

Instrument development . The survey was developed and 4^!iplemented in 

' : ' ^ ^ ■ ' ^ / ' ' ^ ^ 

ytwo stages in order to increase t|ie validity and util'ity of the data and 

/to provide an oppprtunity to refine the Issues of the eyaljuation. asyth?/ 



, evolved. We sent the first survey to- 501 Of the project directors in the 
spring of 1979. At the same time some site visits and the" content analyses 
jof project proposals were comlucted. " The, initial findings from these 

^ pflethods were analyzed together with the responses from the first version 
,of the survey. All results were used to revise the survey whichr-we s'ent*^ 

, ,^to the remaining 50% of CAUSE project/directors in the winter of 1979-80. 

Surveying two. different samples in two stages with an initial anda ^ 
^ revised version greatly increased the power of tfhe resijlting data. 

Issue selection . Xhe first survey was organized in accordance with 
the specific eval uation issues that we had identified as important. These 
Were: 1) the extent to whid/high priority*tocaV col lege and aniversi 
needs are being met by local CAUSE projects; 2) thl ways in" which CAU^ 
projects are benng implemented; S^thg extent to which instructional 
improvement is resulting from CAUSE; 4) the'Daturje and use of evaluation 
data on CAUSE projects"; 5.) program changes and modificatlons to, be made 
in the CAUSE, program. ^ The 5i;cth issue,' the relative costs, of the 
functional activities of CAUSt projects, we thought would not lend itself 

^ well to careful , invest igajti on in a survey. The cost analysis issue 

involves complex questions which must be tailored to the specific project 
being studied. 

Based.,on issues which' emerged from the first survey,' we franed a 
series of questions which were to be expTcfred. in the second version of the 
survey. Having framed the questions, .we constructed items which we hoped 
woyld meaningfully elicit responses, ' . 

, The questions we belleved merited exploration in the second survey ^ 
are as follows; 
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-Are project directors and institutions with experience in , 
writing proposals and managing externally-funded projects more 
likely to write successful proposals and manage successful 
projects than their colleagues without previous experience? 

*. ^ . * ^ 

^-Do project fJirectors belong to a communications network through 
^ which they heard about CAUSE? 

-Is it the case that most successful proposals were submitted " 
and re-submitted? 

-Are organizational issues (e»g», obtaining the commitment and 
codperation" of administrators) important to project' success? 

.-Is there a general confusion .about what NSF tneansr by evaluation? 

-Has there been general difficulty in the articulation between 
proposal and implementation especially in the areas of the 
management plan and the timetabl-e? 

^-Are the most important outcomes in the areas of Improved faculty 
and/or institutional relations, student competency and improved 
\ ^ student attitudes? 

-Are incentives provided by the institution to (^USE project 
staff important to project success? 

These questions could be answered only in terms oi" project directors' , 

perceptions- about what coiitributes to project success or what the most 

important outcomes might be. " ' 

^ Administration ^ Local CAUSE projects are extremely diverse in terms of 

several variables: duration of project (12,^4, or 36 months); type of 

institution (two-year college, baccalaureate granting college, Ph.D. , 

granting univers'ity or a consortium); amount of NSF funding (up to 

$300,000); and academic discipline focus (natural, physical, social 
s. ■ ■ 

sciences or multi-disciplinary). 

We b&lieved, therefore, that it was necessary to survey the total 
population of project directors for three years - 1976, 1977 and 1978 

y * 

(n-201). Of .these, 50% were randomly selected to participate in the first 
version of the survey, and 50% in the revised version* 




In addition to our evaluation staff, outside consultants reviewed and 
critiqued both versions of the Survey of CAUSE Project Directors, Tbe 
consultants >represent a wide^^ range of expertise in the following areas: 
science education research, instructional development and program evalu- 
atiori in higher education, tests and measures development, instructionaT 
improvement projects in science, attitude scaling, technical writing, and 
CAUSE projects* Many helpfi>l suggestions from the consul tants were incor- 
porated into the final version of both surveys*. Several questions were 
dropped based on their suggestions; others were reformated or reworded. 

The first version of the survey was mail-ec! out in the spring of 1979, 
Eighty-nine project directors, completed it. 

Tlje processes of refining the issues to be addressed in the survey 
and of revising the survey took place in the summer of 1979. We sent the 
revised sui^vey to the second sample of project directors in the winter of 
1979-80* .Ninety-five project directors completed the second survey. 

Data analysis * Data from the first version of the su?vey were coded 
case by case from the survey forms and put on punched cards in order to 
create a computer disk data file. Initial analyM^ included frequency, 
range, and variance of responses on a question by question basis* Cross 
tabulations of responses by question with. the key CAUSE^riables " 
length of project, type ^f institution, amount of funding, and aq/denic 
discipline focus were made. Free responses and respo^es to opeYi-ended 
questions were analyzed, through a qualitative data reduction procedure 
involving the categorizing, sorting and sunmarizinj of responses. ' 

Data we had collected from other parts of the ongoing evaluation ^were 
combined and analyzed together witti tne results of the first version of 
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the survey. These data formed the ba; 
examined in the second version of the 

•i 

Da^ta from the second survey were 
as the first version- ^ 



4 

is for determining issues to be 
survey. ^ 

coded -and analyzed in the same manner 



, Summary of^ Findings 

The results of the two^urveys, taken together, are reported here 
for those questions which are coramon^to both surveys or are similar 
enough to Be discussed together. Table 1 por^trays the percentage of 
project directors^/oiineach sample who chose, each alternative to each 
question that wasxommon to both surveys. The questions do not appear 
in full but have been ^ortened to statements which portray the topic of 
each. For V full portrayal .of the percentages responding to each option, 
pleaSe see the tables. 

In many ways the two surveys are not, strictly speaking, comparable. 
Some questions from the first survey were dropped for the second;, some 
questions were added to tfte second survey that were not present on the 
first. Moreover, the first survey was meant to elicit issues and concerns 
from project directprjs which could then be investigated more fully in the 
second survey. Howe^lce!^ there are some questions and issues which remain 
the same in both surveys in-order to establish response reliability. TKe- 
discusMon which follows is organized according to the major issues. * 

Institutional Need ■ ( 

There are a number of potential indicators of institutional need* 
One, i^ whether CAUSE-related activities were initiated under other funding 
before the CAUSE grant was obtained and whether CAUSE activities will 
continue when CAUSE funding ceases. Another indicator is the level of 
comni.1tfnent from administrators and other campus policy makers,. _^ 

.It may,' of, course, be argued that CAUSE activities can, receive 
institutional support^^lfhout necessarily reflecting institutional need: ^ 



a minority of faculty or administrators may have alneed which is not 
general ^for the Institution, They may write a proj)Osal and receive . 
funding, It Is. urrllkely, however, that the Institution would continue 
CAUSE activities after NSH funding ended or that the majority of Institu- 
tional policy makers wpuld-^become committed to the CAUSE project unless, 
there existed a general perception that CAUSE activities satisfied wide- 
spread institutional needs/ Further, it is likely tbat an Institutional , 
history of support for CAUSE-related activities which pre-dates the CAUSE 
grant Itself is strongly suggestive of Institutional need. . - ^ 
■-^We asked the second' sample of project directors whether activities 
on th« CAUSE project were begun before NSF funding was obtained. Fifty- 
y four percent Indicated that, Indeed, CAUSE activitfes were begun before 
NSF funding was obtained. This is a -substantial percentage and indicates 
that institutions were initiating activities meant to be responsive to 
V^reat needs and applied for CAUSE monies to hel^support these activities 
.wKen the, institutions could not bear the financial burden alone, That NSF 
funds were .needed is indicated t^y the percentage of first sample project 
directors (46%) who reported that without CAUSE/funds they would have * 

" * * . r 

qiyen up ^heir projects* ; 

asked the secon'd sample of Rroj^pt directors whether they- 



,.had sou^i^or would seek additional funding when CAUSE monies ran out 



We 

Sevepty percent of project^l44cectors said they would seek additional 
funding. ,0f those who said they had not or would not, virtually all of 
them reported ttiat their institution would support the continuation of 
CAUSE activities. $tPong Institutional-support i^-^uggested by the fact 
that all proiect directors in the second sati^le be! ieve that support will 
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be found for the continuation df CAUS^ activities except in cases where 
CAUSE supported !*one time only" pu*><^f^ases like facilities construction or- 
equipment purchase* * . * 

The cominitment and cooperation of institutional policy makeY'S *s^eins - 
a, bit more probJIematlcaK Virtually all project directors agreed that it 
was an iin)ortant part of the job of project director t& promote good 
coninunication and good relations throughout the campus/ The importance 
of gaining cooperati9n from all relevant groups on campus was stressed 
over and over again, in .different contexts* In the open-ended questions 
project directors cited atkinistra'tive commitment and faculty acceptance 
as crucial to- project success* They also cited these saine factors as 
difficult to obtain. In the forced-choice questions communication problems 
within the ins^titution and the reluctance of important department and school* 
''atiniixistrators to commit themselves were cited as two of the leading diffi- 
culties by both samples* Clearly, gaining the cooperation of interest 
groups on campus is related to the project director's role as an opinion 
leader and opinion maker* It is possible that project directors who cited 
gaining institution-wide cooperation as an issue for them originally 
believeji rather naively that running a CAUSE project meant simply sj((per* 
vising implementation activities.^ 

This' would be in keeping with the finding . that the vast majority of 
project directors agreed that "our project has cooper&tion from our 
institution's administrators at all levels" and that, "in general, our* 
project has received cooperation from our entire academic community". 

If this argument is correct? then it is probably the case that 
institutional policy makers generally support the CAUSE project on their 
wmpus and that the project meets some institutional needs* . 
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Implementation . . 

We asked project directors to describe how their projects were being 
implemented by asking them to respond in three areas:' cjiarig^ they had 
made from originally proposed activities; activities they believed were 
important to project success; difficulties they had encountered during 
the implementation process. * - . * " 

Issues concerning time and deadlines .wfere meritioned in Connection . 
with all three areas. Timelines were most frequently mfent^ioned as com- 
t)onents of the project" that had changed from what was described in tbe 
■proposal. Delays in receiving materials and receiving approval from NSF 
were two of the most freqi/fently cited difficulties. ■ Careful. planning and 
timely implementation were mentioned l^y a large majority as tmportsnt bo 
success. There- has been a great deal of frustrstion over the inability 
of project staffs to.meet deadlines they had set'^ Some cOntrfbuting 
factors might liaye been: naivete (S^ the proposal writers about how long * 
project activities wtU take; NSF's schedule of approval of CAUSE proposals 
which is not consistent with some institutions* planning schedules; changes 
in project staff or -in the management plan at the institutional level. 

It does seem to be the case that an important element of project 
success is a compatible and committed project staff and that a serious 
difficulty, when it happens, is loss of staff or changes in staff » 

Project directors noted the importa^nce of choosing project staff 

carefully* They also expressed frustration over the conflicting commitment 

' '% 
*f 

of project staff* It was felt by several project directors that staff . 
necessa*|ily p.ursue teaching, service, and scholarship activities because 
those are the activities which the institution is willing to reward* As 
a result, work on CAUSE has been an add-on. A related issue iS the 
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perception of several project directors that they h^ve not gotten maj^imum 

produpttvity from- their project staff. i 

'i 

Perhaps "as important to projett directors as comnitment is good ^taff 
ijelations. Over 80% of \?Qth samples reported that working collaboratively 
with project staff is important to success, 

A related issue, tnentioned sponfaneously ,in open-ended questions, i\s 
the role of the project director. Many project directors mentioned the I 
enormous time and energy involyed in performing the job* Apparently it i^ 
more demanding a role than many expected* The neces&ity of commitment 
and dedication was underscored. 

The need for good management skills was also mentioned. The respon- 
sibilities irfvolv^cf in meeting deadlines, allocating responsibilities to 
ottier staff members and being the primary coninunication link to, other parts 
of th§v instijutioft seem to some project directors a great deal heavier - 
thah they had bargained for* - , ^ / 

■project'directors also mentioned their need for iaformatidn ?bout " 
other projects at other institutions. Many believed that seeingSither. 
instructional improvement projects in operation could g^ive them valuable 
assistance in achieving their goals and avoiding possible pitfalls. 

Instructional Improvement ^ 

In both samples there are indications that project directors believe 
'that instructional improvement has taken place* Clearl^y this is the most 
important outcome of any CAUSE project* Ultimately, any improvements 
brought about by CAUSE monies should benefit the stydent* 

If the argument is credible tf»t improving faculty guarantees 
instructional improvement, then two facets or conjpdnents of instructional 
improvement are affected;' studentrf)utcomes and faculty outcomes* 



Although none of the quesjtiAns about project impact are the same 

from Survey One. to Survey Two,, there are a few which try to elicit similar 

information. In fact, the questions on Survey Two are an attanpt to place 

an increased anphasis on impact issues in light of findings from Survey One 
While on Survey One> we asked two forced-choice questions, one on perceived 

changes in student academic performance and the other on perceived changes 
in the quality of instruction by faculty merribers, we asked four open-ended 
questions on Survgy Two; what changes, in students were expected? what 
changes were observed? what changes in faculty were expected? what 
changes were observed? Thus, the kinds of responses that could be given 
were considerably widened in Survey Two; . ^ 

Almost 90% of Saipple One project directors reportgp great positive 
change or sm6 positive change in quality bf student academic preparation. 
Thirty percent of Sample Two project directors spontaneously reported 
having •observed improved stQdent performance and competency as a result 
of CAUSE, Between oner fifth, and one quarter of project directors in both 

samples also reported, primarily in the open-ended questions, improved 

student attitudes toward science an^d greater student participation- in 

decisions and activities affecting their education* 

Finally, a substantial number of ccMnents cropped up from both samples 

■expressing enthusiasm over additional opportunities created for students* 

Whether the comments were in connection- with questions about improved ^ 

faculty performance, new or revised curricula, new facilities, equipment, 

or computer capabilities, or new instructional strategies, they all said ' 

the same. thing,: the CAUSE project has Created new options and new services 

that can enrich the experience of th& student enrolled in CAUSE-related 

scjence courses, . ' * 

-in • . 



Project directors expressed a belief that student^re benefiting 
from CAUSE activities in a variety of ways, some of them related to 
improved feelings of self-confidence and mastery when confronted with 
science materials. 

survey One asked project directors to rate the level of improvement 
in.quality;of instruction for CAUSE faculty members, for non-CAUSE faculty 
members in departments formally involved in the project and by nob-.CAUSE 
faculty in, departments nol^ formalTy involved with the project. Sample^ 
One project directors reported that 85^ of CAUSE faculty have experienced 
som^ positive change or great positive, change; 42% of non-CAUSE faculty 
•in CAUSE departments have; and only 7% of non-CAUSE faculty in non-CAUSE 
departments have experienced positive change. Sample Two project directors 
did not make those distinctions but spontaneously reported that the 
leading change in faculty is that teaching effectiveness has.been enhanced 
and faculty have ^a better relationship with: students (22%); and the second 
most freq.uer|tly cited change ts that faculty capabilities have been 
-broadened anfJ increased and faculty are engaging in new activities (21%). 



In thisjconnection, it is also interesting to note that Sample One \ 
project direcjtors reported in Question Nineteen that th^Tnost important • 
success of'thpir project was the growth of faculty awareness and improved 
attitudes and;skins {zm). 

'^' ^^'^It would seem, then, that whatever project directors describe their 
projects to be primarji^ly about, an important outcome is faculty dtvelopmeat. 
Faculty are learning4)ow to write grant proposals, how to use the computer, 
how to do curriculum development and how to use new'teaching"^5tfategies 
as a result of the CAUSE project at their institution. Project directors, 
reporting this in the o(3gn-pnded questions in both surveys, sounded 
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surprised and pleased by this. > 

Several project directors from both samples also reported, with some 

surprise, that faculty at tf^eir institution and particuUrly in departments 

directly affected by CAUSE were working more collaboratively than expected. 

Many CAUSE projects encourage the development of an esprit de corps that 

was, not present before CAUSE, 

There is also the sense that faculty morale has improved as a result 

of learning new skills and/or being exposed to new domains of knowledge. 

Evaluation ^ 

The^ffrst sample of project directors was asked. many more questions 
about evaluation tharTthe second. In the first survey^we asked project 
directors to report on the present state of their evaluation activities; 
what aspects of the project were being eva^luated; what types of data were 
being collected; and who participated in decisions effecting project ' ' 
evaluation. We also presented a list* of statetR^nts about evaluation and 
asked project directors to agl^ee or disagree with them, In^the second 
survey, there was only one item covering project evaluation;' a list of 
statements {different from the one in Survey One) with which project 
directors were asked to a^ree or disagree. 

Belcause of the differences in the questions between the two surveys, 
it is difficult to compare Sample One wit^ Sample Two on the issue of 
evaluation. There is> however, detectable in both surveys* an unease 
about the issue. Both sampJes reported that the eyaWtion plans described 
in the proposal are not an ajccurate reflection of evaluation as it occurs. 
Project directors in bott^ samples agree with statements which suggest 
that evaluation is valuable^^Ninety-three percent of project directors 
in Sample One agree with this statement from Question Seventeen: "It is 



'important tt^at CAUSE guide]ines'.require eva^iuation as part of projects." 

,' ' ' . ' ; ' ■ • 

Eighty-.slx percent of project^ directors agriee with this statement: 

• ' ■ .. \ , ' V 

"Evaluation plays a mortf prominent role in our project than It does else- 
Where )n our institution'^s science programs/' Sample Two project directors 
agree In ^high" numbers with statements like the following; "Evaluation is 
importan^t to the instftutidn in monitoring the effectiveness of projects 
of this type^' (9356); "Evaluation results have been used to change some of 
the activities and/or outcomes of this project*' (68%). 

^ the same time', a substantial minority in both samples indicated 
an impatience with evaluation activities as being wasteful of resources* 
Twenty-five percent of Sample One project directors agree that the 
evaluation of .their CAUSE project probably re^luires more time and effojrt 
than it is worth;' and 21% agree that formal evalaation activities take' 
too much tim^-^d effort for their project. A large minw^iiy of Sample ^ 
Two proje<rt directprs agree. Twenty-seven percent agree that formal 
evaluation, activities take up too much time, effort, and money on their 

CAUSE project; and 271. agree that project funds allocated for evaluation 
activities could be better spent on other project activities. 

-Ther^ seems to be some (division of feeling about evaluation in both 
samples. It may simply be that project dinectors approve gf evaluation 
in princip;le and believe in its potential usefulness, but do not want 



to e)«pend project resources on evaluation activitiea^^at- they believe 
might be better used for implenipntation* A large miryjrity of both samples 
reported that evaluation. has not gone as planned, arnd both samples reported 
that they perceive evaluation as\ important to project success* Evaluation 
seems to be both a source of ftip^tration and a vehicle for achieving 
desired outcomes* / \ 



Recommendations ' ■ * ' 

Several questions on both surveys asked project directors if they had 
any reconwendations to make' to NSF.' By far the most, frequently recurring 
recommendation is for more information sharing. Several *forums for informa- 
tion exchange were suggested by project directors: NSF s^uld hold meetings, 
edit a newslettV dr find some other means of disseminating information about 
various aspects of running a CAUSE project; there should be sharing of infpr- 
mation among project directors in whi^:h mutual help is offered. More 
experienced project directors might offer a "lessons learned" workshop or 
publication. ' - v 

HaT)y project directors stated that they feel isolated: ^They also 
tend to, learn the hard way how to manage a CAUSE project. There is a 

widespread belief thSt each pro-ject director should nof be required to 

■ ^* 

re-invent the wheeU 

, It is important to'mention that many project directors praised NSF 
suggesting that HSF had always been both helpful and flexible. Comments 
praisiny NSF*s cooperativeness appeared not only in responseto a question , 

sbliciting reconmendations to. NSF in Survey Two, but appeared spontaneously 
in margins and at the erid of the survey where extra coimjents were invited. 

Sample Two pVoject directors also were asked to make recommendations 
to prospective project directors in the areas of strategies for ensuring ■ 
project success and for evaluation of project activities. These Inay be 
considered a list of lessons project directors have learned from their \ 
experience on CAUSE projects. 

The three most commonly mentioned strategies for ensuring project 
success' are: Kl) making sure management .and authority Issues are worked 
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out within the .project staff; that is, being clear and ^finite about who 

does what, when, and who reports to whom; knowing who is doing what for^Jtbe/ 

sake of effective monitoring; (2) doing careful front-end pi annl3r\g ; '♦trying 

to foresee what problems might arise and being prepared witFi, contingency 

: ^ : ^ \ ^ ' . 

plans; and (3) getting the coninitment of the faculty to the project"; making 

sure there is widespread support for project jartivitjes,^ 



The three most conmonly mentioned recotmendations concerning proj(K:t 
evaluatioh are the following: (1) using outside experts to do^the evaluation; 
these may be from oiitside the institution, but should at least .be^fpom 
outside affected departments; (2) being realistic about evaluation;' not 
trying to accomplish rrore than is reasonable.^ finding an evaluation plan ^ 
that the project director can live with and be satisfied with; and (3) dpinq 

. ' — ^ , \ ' ' ' : I ^ ^' 

a formative evaluation in which modifications are made^aj the project', - 

' . . ' ' ' 

evolves in -light of Information that has been gathered, T ^ 

* , .. * x . \ ' , '' 

Thes^, then, constitute the general findings of the two^uJ*Veii^ , 

taken together. The fojlwtng pages provide a detailed discussion V the 

findings from each question from each version of the sijrvey»j v , ' 
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Table 1 

Comparison of Responses to Questions ^ 
WhicH Appeared on Both Survey One (S-j) ancf Survey Two (SgJ 

Percentages of Response 



Question / 2a, (S^) 
Question/ 12a, (Sg) 



Accuracy^ of the original description in 
the proposal. for each of the. following 
project' components. 





Accurate 


Inaccurate 


■ No Response 
^1 ^2 


Project objectives 
and goals 




99 




, ,1 


• 0 0 


Project management 


. 97 


97 


3 




. 0 1 


Timetables or time^. 
lines * 


79 




■ 20 ■ 


33 ■ 


1 ' 0 


Evaluation plans' 


87 . 


.82 , 


■12 


17 


f 1 


Impact of project^ 


' 87 


88 ' 


' . 8 


6. 


5 ' '5 



Question 5a, (S^) 
Question 13a,JS2) 



Importance of som^ activities to 
project success. 



ERiC 



Important 
?l"! h 



Unimportant 



paperwork 



Doesn*t Apply or 
No Response 



1 



Project planning and 
■ management sessions 




82 




.14 


0 ' 

t 


2 


"-Efforts to win support 
for our project at 
our institution 


69 

. i 


78- 


2& 


18 


11 


/ 4 


WorJcfng with faculty' 
members on the * 
project staff ^ 


;94 


83 




13 


3 


4 


Working with students 
on the project , 


66 


70 ^ 


27 


22 


7 


8 


A<jvising students . 


48 


. 32 


25 


40 


27 


.28 


Writing rfeporj:s and^ 


. 46 


39 


51 ' 


59 . 


3 


2 



22 



Doesn't Apply or 



* 


Important 
^1 ^2 


Unimportant 


No Response 
^1 ^2 


Ordering supplies 
Qnd equipment > 


78 


74 


21 


23 


1 3 


Evaluating the project 


90 


83 „ 


9 


16 


1 1 


u€s ign 1 ng jnsLrucii una i 
materials . ' 


92' 


88 


4. 


5 


^ D 


ucsi gni ng laQfl n t les 
and selecting 
equipment / 

; / 


89. 


8.4 


, 6 


9 




Describing^ the project 
to other^ 


69" 


69 


28 


28 


. 3 ' 2 


ueve J op 1 ng a new 
CHjrrlculum 


61 


58 


20 


24 


1 y * 10 


^ecKi ng in nsnci si sup- 
, port for the project 
once NSF funds are gone 


63 


60 


' 30 


28 ^ 


fix 7? 


Teaching related to 
our project 


96 


92 


3 • 


4. 


1 4 


Horking with lab 
technicians 


.38 


56 


30 


21 


31 25 



Question 4a » (s^) 
Question 15a. (s„) 



Seriousness^ of various difficulties 
which may arise on a CAUSE project. 



Serious 



Not 
Serious 







^2- 


^1 


^2; 


Si 


Delay of formal approval 
\,of .our project by NSF 


8% 


13 


83 


68 ' 


8 


Confusion of responsibil- 
ities wit^iin our project 


4 


3 ^ 


84 


2 


Insufficient attention . 
given ,to project planning- 


5 


2 


91 


76 


, 3. 


Unclear decision making • 
policies pn our project 


3 


2 

t 


93 


79 




Lack of necessary techni- 
'cal assistance (i.e-. lab 


7 


.12 


89 


67- 


4 



Don't Know or 
No Response 

^2 

2 

. 0 
3 
2 
2 



Doesn ' t 

^2 
17 

12 

19 \' 

17 

19- 



.production. A-V equip- 
* ment, etcv) 

Short supply or ctelay, 
of materials 



14 83 74 



12 



Not Don't Know or . aoe^nit 





Serious 


Serious 


No Response 
^1 h 


Apply. 


Conmunicdtion problems 
within our institution 


n 


14 


87 


76 


2 


3 


7 


Mis understand inj^of 
project objectives by 
project personnel 


2 


5- 


96 


82 


2 


2 


11 


Reluctance of important 9 
departnient. or school ad- 
ministrators to commit 
themselves to our project 


16 


87 


75 


4 - 


• 2 


. 7 


Lack of attention given 
to problems of imple- 
mentation by project 
personnel 


3 


n 


65 


78 


4 


2 


10 


Conflicts among project 
personnel 


6 


3 


91 


80 


3 


2 


15 


Difficulties with our 
. institution* s rules 
and regulations 


3 


13 


93. 


74 


2 


2' 


12 


Difficunies vith -NSFls 
rules 'and regulations, ^ 


0 


0 


98 


81 - 


2 


■ 2 


11 



,Sp- K=89; Sg* N=95, 

Kote; Complete results for these questions appear in Tables 2 & 3, 

^**Accurate'* represents the total percentage of project directors who chose 
.the Options, "generally accurate or very accurate in the oflginal question* 

"Inaccurate** represents the total percentage Qf project directors who chose 
the options > generally inaccurate or very inaccurate* 

/ /V 

^Vlmpbrtarijt** represents the total . percentage of project directors who chose 
tl^e options^ extremely -important or important, 

"UnimpprtanJ'* represents the total percentage of project directors who chos 
tW options* H^omewhat unimportant or totally unimportant* 

"Serious** rtplre$ents the total percentage of project directors whoychose^h 
options* Critically serious qr serious, * > 

*'Not Serious!* represents £he total percentage of project directors who chos 
the opti9ns* somewhat serious and not serious at all, 

*^TKis category was used only in Syrve^Two, . ' 
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, .Table 2 

Survey of CAUSE Project DirectoFS 
Percentages of Response 
Spring, ,1979 
. N « 89 



1. Extent to which. project is meeting or Wi.ll meet its original goals 
as stated In the proposal . ' 

Completely Partially • OriTy Slightly Mot Achieved No 

Achieved Achieved : Achieved -M All Response 

- . 57% 40 1 . 0 1. 

2a. Accuracy of the.orig'inal .description, in the proposal for each of the. 

' following project componenty^ ~ ^ . ■, ■ , 

Very Generally Generally Very No' 

Accurate Accurate Inaccurate Inaccurate Response 

Project objectives 62% " .36, 1 /O ,^ 0 . 

and goals ' ■ 

Prbject management' 41 54 2 1 0 "" 

Tifnetflbles or ^ "6 71 17 .3 . 1 • 
timelines ; 

Sudget ^ .20 73 ' . 4" ' 1 0 

■ * ' ■ > 

Evaluation plans ' 15 ; 70 12 1.. 0 

Impact of project 33 53 6 1 * 5 _ 



Npte: Percentages may not add up to 100% due to rounding error or 
because project directors were free to give more than one. 
response. Questions which appear'in italic type had open-ended 
responses which were then categorized^ For these questions per- 
centages aVe shown for both the total number of survey respon- • 
dents and the number of respondents to the question. 
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2b', HcM has your project been rpdi/i^d during its operation to incorporate 
new findings and/or experience gained ? 

S£ Of S£ Of ' * ^ 

Total Respondents 



OA 


OR 


rfoposeo acti vi 1 1 es/iTidndQenient p i an 


1 0 

' 1 y 


01 ^ 

C i 


tonstruction/eqinptnent 


18 


1 /I 

19 


No changes 


17 


18 


Schedjile/timetable 


10 


11 • 


Budget * 


9 


10 


Project staff 


; 6. 


.6 


Different Courses 


6 


6 


Evaluation 


4 


5 


Goals/objectives 


' 4 


5 


Materials acquisition 


.1 


1 . 


Reward structures for participation 



jopera 
acuity 



faculty iwmbers. 



3, Cooperation receiv ed from the* institution's administration and 

* ' Doesn't Apply 

Strongly Strongly err 

Agree Agree Disagree Disagree No Response 



Our project has co- ' 60% ; 31 4 - 2 . ^ ^ 

operation from our 

institution's adKiin- , - - 
istrators^at all levels 

Our project has co- 69 30 ^0 0 1 

^ operation from all- ' ' . ' 

, faculty members who 
. are part of the CAUSE , . - - 

project "staff , ' ^ 

Our proiect has co- 25' * 45 15 1 . * 14 

operation from all ' ^ - 

^ non-CAUSE faculty ' ^ / 

members who are in 
* CAUSE project de- . v 

. partment^s) . , * \ ^ */ 

Our ^project has co- '16 ^35 10 ^0 ^ * ^9 

operation from all . - ' — , 

non-CAUSE science . . 

faculty members in -non- . 
CAUSE departments . ' . . 

In general r our project >^ 33 47 7 2 11 ^ 

has received coopera- * ^ ^ 



tlon from our entire 
O acadanic community 



ERIC 



^ 4 
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4a. Seriousness of various 4iff1cu1tie$ which jnay arise on a CAUSE project . 

NQt Doesn't Apply 
Critically Sbmewhat At All or 

Serious Serious Serious Serious No Response 



Delay of formal ap- 
proval of our pro- 
ject by HSF 


1% 


7 . 


18 


65 


8 


Confus-loh^f respon- 

SlDlllLiea WlLnjp 

our project* 


1 


• 3 


• 12 


81 

A 


2 


insuTTic len t avx^niipn 
' given to project 
planning 


C 


0 


1ft 

1 o 


■ I 


0 


Unc1atf^ decision maki^S 
policies on our project 


2 


1 


12 


81 


3 


nical assistance (i»e»> 
lab ass.istance» materials 
productionjr A-V equips 
*inent> -etch . ^ ^ 


c 


4 


16 


73. / * 

I 


4 


.materials 




13 


34 


49 


2 


Cprnnuhlcation probl^dns ^ 
within our institution 


2 


9 


26- 


61 


2 


Misunderstanding of 
project objectives by 

nrni Art nor^nnfipT 


1 


1 


18 

• 


78 


2 


Reluctance of important 
department or school 

V aunnni 5 ira Lur^ to corn" 

mlt themselves to^ our 
project 




6 


20 


; 66 ' ■ 


4 


Lack of attention given 
to problems of imple* 

^mentation by project 
personnel 


0 


3 


29 


63 


. 4 


Conflicts arpong project 
personnel 


1 - 


4 


12 


- 79 . 


3 


Difficulties with our 
Institution's rules 
" and regulations 


3 


* 


22 


. 71 


2 


Difficulties 4lth NSF*s 
rules and regulations 


0 


0 • 


3 


94 • ' 


2 
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4b* Are there other difficulties ^ou have encountered in -project 
implementation which we have not described ^bove ?^ 

' 9t Of % of ^ ^ 

Total / Respondents ^ 

No, nothing serious 

Tinietdble could not be adhered to 

Too little support from institution 

Changes in program from, proposal ^ 

Institutional budget inadequate for full 
implementation 

Loss/changes in staff 

^ Problems in promoting program to students 

Project staff had too little time/energy 

Too much responsibility of director 

Problems with non-NSF state/federal agencies 

Poor communication NSF 

Difficulty in manageillent of project 



19 


27 


12 


18 


11 


16 


9 . 


13 


7 


10 


-4 


■' 6 


3 


5 


3 


5' 


2 


3 


2 ~ 


3 


2 


3^ 


2 


3 


2 


-:.3 



Evaluation ^ 
5a» Importance of some activities to project success . 

Doesn't Apply 

£xtreme1y Somewhat Totally or 

Impor'tant Important^ Unimportant Unimportant No Response 

Project planning ^5% * 39 16 0 0 

anjl manajjement 

sessions \ , 

Efforts to win 35 34 13 7 11 

support for our ■ ' . , 

project at our ' . ' « ^ 

institution 

Working with f ft- 53 . « • 2 0 3 

culty members on ^ , . 

the project staff 

Working with, stu- 27 ^ 39 > 25 ^ 2 7 

dents on the ' " ' ^ ' 

pro;tect . ^ ^ 

Advising students 17 ' 31 19" 6 ^ 27 

Writing reports and 3 43 44 7 3 

related adhiinistra- . - .'^ 
tive paperwork ^ . ^ , 



^4 



Extremely 
Important 



28 



^ Doesn't Apply 

I Somewhat Totally or 

Important Unimportant Unimportant No Response 



Ordering supplies ' 
and equipment 


39% 


38 


19 


2 


1 


Evaluating the 
project 


27 


63 


9 


0 ' ■ 


1 


Designing instruc- 
tional materials 


61 


31 


4 


0 


3 


Designing facilities 
and selecting * 
equipment 


58 


30 


6 


0' 


6 


Describing the pro* 
ject to others 


17 


52 


28 


- 0 


3 


Developing a new 
curriculum 


36 


25 


15 


4 


20 


Seeking financial 
support for the * 
proje^;t once NSF 
funds are gone 


26 


37 ' 


■ 22 


8 


6 


Teaching (related 
to our project) 


58 


37 


3 


0 


1 


Working with lab 
technicians 


18 


20 


21 


9 . 


31 



5b, Are there dther activiti&s not identified above that are important 
to project success ? 



% of 
Total 


% of 
Respondents 


* 




6 


15 


Dedication/commitment of all involved 




3 


9 ,' 


Institutional/community support . 




3 


9 


Good planning/goal setting 




3 


9 


Getting extra funding or, finding ways to make 
do with present level 




2 


6 ■ 


Good job of hiring staff , ^ ^ ^ ^ 




2 ■ , 


6 


Interpersonal dynamics 


' t 


2 


6 . 


Mechanics of producing materials/student use 
of materials 




.2 


6 


Evaluation 




.1 - 


3 ' 


Reviewing instructional materials 





ERIC 
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6. Percentages of students served by CAUSE projects who are science 
majors or non-science majors . 



51 "of students f 

0 , 

1-9 
10-19 
20-29 
30-"?9 
40-49 
50-59 
60-69 
70-79 
80-89 
90-99 

100 



Science Majors Non-science Majors 

of projects % of projects # of projects % of projects 

45 . 

11 
6 



Total 



31 
6 



35 

7 



10 



10 

n 



89 



101 



40 
10 
5 
. 8 
2 
4 
4- 
1 



9 
2 
5 
5 
1 
3 
3 
6 
5 



89 



101 



7, Extent of involvement of different groups in CAUSE projects , 



Extensive Some ' , Minor No^ No 

Involvement Involvement Involvement Involvement Resoonse 



Institution admin- 
istrators- \ 

Department heads 

Faculty members 
on the CAUSi . 
project staff 

Faculty members 
in CAUSE de- \ 
partment(s) \ 

Faculty member*[ in 
non-CAUSE de- 
partments 

Students ■ • 

Evaluation experts 
Media- specialisti 
Lab technicians * \ 
NSF staff • \ ' 



7% 



25 
93 

38 



44 

17 
13 
22 
1 



38 

39 

7 

40 

15 

36 
51 
19 
16 
15 



46 

27 
0 

17 

45 

13 
24 
24 
16 
52 



8 

8 
0 



38 

3 

9 
43 
45^ 
33 



1 

1 

0 



3 
0 
1 
1 
0 



.30 

i 

J 



8. Funding^alternatlves which might have been taken If CAUSE funds 
had not been available . 

' 13J Sought other federal funding 

2 Sought state funding 

9 Sought private funding 

3 Sought additional resources elsewhere within our institution 
3 Undertaken the project on departinent(s)*s existing budget 

46 Given up on the project for lack of funds 

23 Other 



9. Direction of changes which might be related to CAUSE project activities . 

Great Some Some Great 

Positive Positive No Negative Negative No 
Change Change Change Change Change Response 

Quality of academic 
preparation of stu- 
dents attributable 
to our CAUSE project , 

Quality of Instruction 
by CAUSE faculty 

Quality of instruction 
^'by noti^^AUSE faculty 

meirt)ers- in departtnents 

formally Involved with 

our project 

Quality of instruction 
by non-CAUSE faculty 
members in departments 
not formally involved 
with our project 




10. Innovativeness of CAUSE projects as compared to : 



Very Somewhat Not Not No 
Innovative Innovative InnovatWe Sure Response 



Regular activities of the 
clepartment(s) Involved ir^ 
CAUSE 

Other science departments 

Science depd'rtments ' 
nationally 



52% 

43 
24 



42 

38 
48 



3 
6 



5 . 
21 



Ti 
1 
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10b» Please id4ntifg briefly the most innovativ^^-^ilBpect of your prajject * 



% of . 


% of 


/ 


34 


35 


Unique courses or aspects of courses 




20 


Computer use 


n 


• 12 


Bringing together faculty to work together/ 
Treeing up Tacuity lor stuoents 


10 


11 


Involvement of students in aspects of the project 


9 


9 


Individualized instruction 


8 


8 


Changes in majors- and/or sequences af courses 


6 


6 


Evaluation of program 


2 


2 


Equipment and laboratories 


2 


2 


Use of institutional personnel other than 
faculty 


1 ' 


1 


Relationship with outside agencies 


1 


1 


Introduction of media center 



11a, Have science projects (either CAUSE or non-CAUSE) similar to yours 
at other 1'hstitutlons been a useful source of Information and Ideas ? 

575f . Yes 30 No 12 Don't know 

lib* If they feaVe h^n, hw did you learn about them ? 



% of 
Total 


% of . 
Respondents 


1 r 


27 


42 


Journals, meetings, books, directories^ 


18 


■ 29 


Word of mouth/personal contacts 


18 


29 


CAUSE directors' meeting in Washington, D»C. 
and/or other NSF meetings 


11 


V ' .18 


Visits to other campuses and/or correspondence 

r 


7 

2 


11 
4 


Copies of CAUSE proposals 4 ^ 
CONDUIT ' ^ 


1 


2 


Chautauqua short courses 


1 


2 


Consultants 


1 


2 


Staff members* ex-NSF readers 



ft 

33 
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lie. If they have not b&ent why were they not useful ? 

% Of %of .. : . p 

Total Respondents 

18 67 Haven't identified any prq^rams sufficiently 

similar 

9 33 Knowledge of ottjer programs has not been 

needed arid/ or ^de available 

\;. / 

12a / Current status of project evaTg&tion activities ^ * 

2% We have not yet begun to consider'' evaluation activities. 

) We have begun evaluation planning but have made little progress' 

18 ^ We have begun evaluation planning and have made modest progress 

1 Evaluation planning is nearly complete* 

0 Evaluation planning is complete, 

71 Evaluation activities are going on now on our project,, 

1 Evaluatiorv will probably not be a part of this project,. 

12b, Evaluation data are being collected on a regular basis already * 
85% Yes 10 No 3 No response 

13* Aspects of the project to be evaluated * (Projects may be collecting 
data on more than one*) ' 

89% Student reactions to project 

78 Student perfomrance 

52 Classroom and teaching processes 

43 Faculty performance 

80, Instructional materials 

55 Courses or curriculum ^ 

72 Project activities .as a whole 

10 ■ Others ^ * " 

« 

14, Measures of student achievef!>en_t which are'part of project evaluation , i 
(Projects may be collecting data on more thaaone*) 

51% Multiple-choice or essay examinations 

17 Papers or essays ^, . ' . 

35 Experiment orOaboratory reports , 

26 . Grading of in-class perforinance i ^ 

40 Overall cojirse grades 

15 Special project grades 

^ 29 Proficiency tests of special skills or special training 

13 Presentations 

6 None ^ ^ ^ 

17 ' Others ' ^ ' 
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15» ■' Other types of evaluation. data . (Projects may be collecting more 
than one) ' . ■ 

71$ Faculty opinions or ratings of project activities or outcomes 
82 Student OBinions or ratings of project activities or outcoines 

34 Observations of students in class 

27 0bservati6ns of faculty ^teaching 

80 ^ Documeijtation of project activities 
49 Interviews with project participants ' 

26 - Attrition reports - , 

'42 Enrol Iment records 

0 Non^ 
12 Other - 

16* Participants in major decisions on project^eyalua'tion . 

* 

16$ The project- director ' „ - 

' 6 A single person responsible for conducting , the ervaluation 
■ (other than the project director) ■ ■ 

9 A small group of project staff '(other than the above) 

40 All or most of project staff 

0 Non-CAUSE. faculty members * ' , 

0 Non-CAUSE - a*iini strati ve personnel 

29 Other - ■ ■ ' - ■ 

' 17» The role of project evaluation ^ 

Strongly Strongly No 

. \ i - ■ Agree , Agree Disagree Disagree Response 

Evaluation plays a more prominent 44% 42 15 0 0 . 

role in our CAUSE project/than it 
does elsewhere in our institution's 

science programs , ' . ^ 

The evaluation of Our CAUSE prOr 6 19 62 ' 13 0 

ject probably requires more time . ^ ' \ 

and effort than Xt 5s worth ' . ^ 

It t$ important that WUSE guide- 33 " 60 6 ^2 0 
lines require evaluation as part ^ . . ^ 

of projects ■ " • , 

* ■ * # 

Project staff have acquired'dd- 24 49 20 2 

ditional expertise in evaluation * 
as a result of .the CAUSE project 

If CAUSE guidelines had not re- - 12 20 53 ' 15 
quired evaluatfoN it would not 
have been. included in this project 

Our CAUSE project has helped 10 33 45 ' 7 

science faculty members to inte- ■ 
grate evaluation into ongoing. 
' science programs at our institu- 

9^- tion V ^ 
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Strongly Strongly " Ho: 

^ Agree Agree Disagree Disagree Response" 

Formal evaluation activities ^ 5% 16 - 62 17 .-1' ^ 

take too much time and effort 

for our project / * ' ^ 

Our CAUSE project has led to an 8 40 ' 44 -3 5^ 

increased concern for the-quality ^ * - 

of evaluation efforts in iry . ^ ; 

department 



18 ^ Zf there are any formal or informal evaluation aQtivities on your * 
" project which have Jiot appeared in the above items/ please, /^^scribe 



these activities below, • ^ , ^- 


.% of 


% of 




Total ■ 


Respondents 




g 


50 


Formative evaluation involving faculty feedback 


3 


19 


Students involved with evaluation effort 


3 


19 


Use of outside consultants • . ^ 


1 


6 


Pre-test, post-test on student achievemenf 


1 


6 


In future, evaluate alternartive niodes of A.V. 






presentation J^' 


what has been the most important success your^project hAs experienced? 


% of 


% of , 


* 


Total 


Respondents 




2a 


29 . 


Faculty awareness, improved attitudes , new skills 


26 


27 


Curriculum development/expansion 


20 




New "facilities/computer 


ir 


17 


Improvement in student attttudes/performance 


11 


12 


New sense of[ conmunity/purppse ^ 


8 


, 8 


Development of new instructional methods 


5' 


, 3 


Development of individualized instruction 


2 


■ 2 


Generation of evaluajtion scheme , ♦ 


2 


2 


. Outreach to other institutions ^ departments 



or industries that are in the forefront of 
I science and science teaching 
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20. What has been the most significant dlsappointmsnt or failure your 
project beis experienced? 



Failure to get maximum productivity. from staff 
Behind schedule 
Lack of institutional support 
Insufficient funding 
No disappointments yet 
Difficulty in gaining student support 
Some project goals not met 
Integration of innovation into curriculum 
Equipment problems 
Evaluation 

Lack of skilled progranming 
Loss/change of staff 

21 . please list any particular' aspects of the CAUSE program tbat you 
believe merit additional study * . 

% of % Of 

Total Respondents 

4- 

9 18 Study successes/failures— do a "lessons learned*' 

dissemination efforjt 

7 1^ Desirability to expand project to new 

populations/departments/fields 

'6 n Do a study on effectiveness of one pedagogic 

innovation over arfOther or over regular program 

6 11 ■ Find a way to do better evaluations 

3 7 Look ^or better managert^t strategies 'for 

project . ^ 



% of 


% of 


Total 


Respondents 


• 16 


17 , 


15 


16 


15 


16 


9 


10 


• 8 


9 


8 


9 


7 


7 


6 


6 


6 


6- 


2 


3 


2 


3 


2 


3 
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Table 3. 

Survey, of cXuSE Project Directors 
PercentageKof Response 
Fall, 1979 
N«95 



la. The three most Important planned outcomes of tthe CAUSE project . 

81% Curriculum additions/revisions 

'59 . ■ Equipment and faciUtles acquisitions 
■ 54 Individualized Instruction . 

4$ Computer applications 
28 ^ Facu}Ay developtnent 
25 Remediation . : 

lb . Which of the six CAVSB project outcomes i>est descrilHts your project? 



% of 


% of 


V 


Total 


Respondents 




28 


' 3l . 


Curriculum additions and revlsjions 
Equlpflieht and facilities acqul/sltlon 




25 


17 


18 


Computer acqulsltlon/appllcatj^on 




17 ' 


Individualized /Instruction / ^ 




8' . 


Faculty development , i 


4 - 


4 


Remediation ^ 



2. ' History of CAUSE activities . • / * 

10% some of the activities on our CAUSE project were begun 
under support from another externally-funded project. 

44 Some of the activities on our CAUSE project 'were begun 
on funds from our Institution's budget. 

40 No activities for this project were begun before the 

preparation of the propositi. 
6 More than' one response. 



Note: Percentages may not add up to 100% due to rounding error or 

because project directors were free to give more than one response. 
Questions wh16h appear l/i Italic typtf had open-ended responses. 
• which werp then categorized. Percentages are shown for both the ■ 
total number of .Survey respondents and the number of respondents 
to the question.^- — ■ 
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3, Institution submitted a proposa1(s} to CAUSE before this one was ' 
. funded. 

■ 38^ Yes, another iifersion- of the current project proposal 
was submlttea* 

5 Yes;, a proposal for another project in the same discipline 
as our funded project was submitted. 

17 ' YeSi a proposal for another project in a different 

discipline was submitted. 
3"4 Nb> the proposal for our current project was the orjly 

one our institution has ever submitted to CAUSE. 
12 No, not to knowledge. < 

4a. Reviewers? coninents requested and received on proposals that were not 
funded > 

45SJ Yes, the reviewers' comments were requested and received. 
48 . The reviewers* connents were requested but never received. 
0 No, the coninents were not requested. y 

6 No, not to my knowledge. 

4b. If thg reviewers* comments were received,^ how wer^ the coirtnents used 
in preparing another CAUSE Pto^saI ? 

« Of 5t of 

Total Respondents 

24 33 Reviewers' comments used to rectify 

deficiencies in earlier proposal 

23 32 This item is 'not applicable 

12 16 Reviewers' comments used to change the 

emphasis of the proposal 

10 13 Reviewers' ci^ittcisms were answered in 

proposals and/or unhelpful comments were 
ignored 

5 7 Reviewers' comments were used to Change the 

evaluation strategies 

3 4 ' Reviewers' comments were used to increase 

Institutional support of the propotal 



5a, Did you participate in the'- development of the proposal for your 
•CAUSE proJgjEt ? ] 



97% Yes 3% No 



33 
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5b , If you answered "yes", bow did you find out about the CAUSE program ? 



6, 



% of 
Total 


-X of 
Respondents 


\ 


38 


39 


Of^Fice of research service, sponsored programs, 


28 


* 29 


Vi < i tni*< f t^Ain N^F ni^ N^F hi^^*hiji^ft< f 1 t*< 


26 


' 27 


College administration 


14 
4 


' 14 
4' 


NSF briefing meeting' 

Prior experience With NSF I 


3 


3 


Faculty contacts 


3 


3 


Faculty went out and lodtced for grant 
^^p^rtunities 


GpOUQ.( 


s) or individuals ''primarily responsible for the development 


of the 


proposal , 


9 



ss% 

28 
,15 
12 
1 



Faculty -croyp 
One faculty menter 
Administrators 
Other f 
Students ' 



7a, Was more than one department or group of faculty interested in 

— — — 



7b, 



applying for a CAUSE grant? 
48% Yes 



53% No 



If you answered "yes", how was it determined whic^ departmnt or 
groups would- submit? 



% of % of 
Total Respondents ' 



.18 


28 


15 




14 


21 


1.0 


16 


5 


^ 8 


5 . 


8 



Cooperative combined effort *^ ^ 

Critstia were established for determining* 
the bfefi^t proposal 
Administrative decision 
Notapplicable or don't know ' 

Submitting gro"up was ready with a proposal 
first ^ . , 

finyjOne interested could participate 
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gov were staff members selected to be involved in the CAUSE project ? 

« of « of ^ * 
Total Respondents 

33 . '33 Selected by subject area, expertise, teaching 

area, or position 

28 29 Self-selected: anyone Interested could 

participate i 

23 24 Selected by prKident, dean, department Jjead 

or projeft director 

16 16 Selected because of proven coimiitnient and/or 

participation in proposal writing effort 

t ' ' ' 

3 3 Item not applicable 

3 *^3 Those who thought the question referred to 

proposal writing activity ^ 



Project director's previous experience managing externally-funded 
projects 1n a higher education settings ^ 

AZ% ' Have managed^ at least one eKternally-funded instructional 
' improvenient project prior to CAUSE. 

39 Have managed at least one externally-funded research . 

project prior to^CAUSE. ^ , ^ 

31 This CAUSE project is my first expef^ence with project 

managemeflt. 

* ■ ■ . . - i 

Previous experience of project staff with Instructional Improvemerit 
projects . 

lis ' None of project staff has prior experience with 
Instructional improvement projects similar to the 
CAUSE project. 

•42 Some of our project staff have prior experience with 

insyuctional Improvement projects similar t» the 
CAUSE project. V k 

46 Most of our project staff have prior experience with 

Instructional ImproVement.projects similar to the 
CAUSE project. 

11 More than one response-. ' 
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11 , If yoa had been able to- regoest additional ovtsMe assistance 
planning your CAUSE project and preparing youx CAUSE project 
proposal / what kind of assistance might have been helpful ? 



% of 
Total 

30 ' ' 
'18 

18 

10 

.8 
7 
6 



« of 
Respondents 

. 32 
20 



Didn't. need any help 

Needed help' in organizing activities, staff 
time," budget 

20. Needed to see successful* proposals and/or 

CAUSE projects, project staffs' 

10 Needed outside consultants in areas not 

mentioned in other categories - 

9 Item not applicable 

8 *^^eeded help on .evaluation 

7^ Needed help in understanding NSF policies/ 

guidelines 

7 Needed help 1n proposal writing ^ 



1 2a . ■ Accuracy of the original description in the proposal for each of 
the following project components. 



Project activities 

Project obj'ectrives 
and goals 

Project management 

Time tabVes or 

timelines 

Budget 

Evaluation plans 
Impact of project ' 



ig project compon 

Very 
Accurate 

55% 

58 



43 
12 

19 
22 
34 



Generally 
Acc urate 

m 

^ 41 V 
54 

At 

56 



65' 
60 
55 



Generally 
Inaccurate 



Very 
Inaccurate 




No 

Response 
0% 
0 

1 
0 

0 
1 
5 
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l2b. h6» ^as your project been modified during the operatiop to incorporate 
hew findings and/or experience gaijied? 



% Of • 

Total 



% of 
Respondents 



26 

- 


28 


Ko changes or slight changes or changes 
as planned 


19 


20 


Changes in implementation strategies, 
methods or activiti€fs 


.13^^, 


13 . 


Changes in personnel or in staff roles 


13 


■13 


Changes in timelines 


12 


12 


Modifications in ^uipment/facilities 


12 


12 


Changes in software, materials 


7 


8 


Changes in budget 


7 


8 


Chansies in computer equipment 

• • *■ 


5 • 


6 


. Changes in goals or objectives 


5 . 


6 : 


Changes in evaluation strategies or plan 



13a. Importance of some activities to project success , 



Project planning 
& managetnent 
sessions 



€xtremely Somewhat . T<JtaHv_^esn't Apply. 

Important Important Unimportant JJn importing or No Response 



32% 



51% 



•1|« 



2% 



4% 



£fforts to- win 
support for our 
project at our 
institution 

•Working colla- 
boratively with 
project staff 

Working with . 
students on 
the project 



44 



45 



19 



34 



38 



51 



15 



12 



, / 
19 



13a. (Continued) ^ < ' 

Extremely Somewhat ■ , Totally Doesn't Apply 

Important Im'portant Unimportant Unijnportafit or No Response 

Evaluating the 

project 26% 57% 14% 2% ^ , 1% 

Designing . ' " . 

instructional 

materials 58- 31 4 . 1 ' . 6 ■ 

Designing 
facilities 4 . 
selecting 

equipment 50 34 , 7 1 '8 \ 

Describing the 
project to' 

others 17 53 25 ■ ' * 3 -2 ■ 

Developing- a ■ ^ 

new curriculum 20 38 , . 21 3 18 . 

Seeking financial I, 
support for the , / 

project once NSF . - . . ' 

funds are gone 20 40 ' '21 ' 7 ' .' t2 ' . 

Teaching (related 

to our project) 42 SCT ' 4 0 4 

Woiiiking with 
Ta& technicians/ 

prbgranimers , etc. "13 41 . 15 - 6 25 ^ 

Working with non- 
project .facMlty 21 25 ' ' 22 .14 .18 
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13b» Are there other activities not identified above that are^ijnportant 
to -project success? 

• S! Of 3t Of , ^ 

Total Respondents^^ ' 

18, * 32 ComnuMcation/promotion of good relations with 

NSF, institution, students, dissemination- 

17 ■ 30 No other activities 

U 19 Setting and keeping cooperation/participation 

of faculty and administration 

5 9 Setting a good, committed staff 

3 6 Use .of'Outside consultants , 

14a. Have you described your project to someone from other institutions? 

90% Yes 956 No n No response 

14b* If gou did have the opportunity to describe gour CAUSE project to 
soineone kt other institutions ^ what- dj^ you say was the most 
important outcome of your project? 

S! of ■ 3t Of 
Total Respondents 

24* 27 " Improved curriculum/upgraded program 

* 19 21 Development of improved attitudes toward 

computers, computer-related materials 

14 15 ImproVed'equipment/facilities 

* ^ , 

, , . 14 • .15 Improved instructional' options/ for students 

' ' * .* • ' 

12 - ; 13 Improved lab opportunities 

*. , 10 11 / , Accommodations to student learning needs, 

■* ■ , "especially non-science majors or slowet 
' . . learners 

10 11 ' Improved student attitudes- 

8 ^9 - Increased student learning 

8 , * 9 * Impraved' faculty attitudes- 

5 6 . No one roost important outcome or not 

a|!iplicable or don't know 



Upgraded or new faculty >^kills 
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15a » Seriousness of various flifflcultles which may arise on a CAUSE project s 



Critlcafly 
' strlous 



Delay of formal approval 
of our project by NSF 

Confusion of responsibi- 
lities within our project 

Insufficient attention 
given to project planning 

Unclear decision-making 
policies on our project 

Lack of necessary techni- 
cal assistance 

Short supply or delay 
of materials 

Conmunlcatlon problems 
within our Institution 

Misunderstanding of 
project objectives by 
project personnel 

Reluctance of Important 
departmeht or school 
a&nlnlstrators to commit 
themselves to project 

Lack of attention given 
to problems of Imple- 
mentation by project staff 

Conflicts among project 
person ne^ 

Difficulties with our 
Institution's rules 
and regulations 

Jifflcultiesi with NSF's 
^les' and regulations 

Lack of sufficient time 
■ to completje planned 
activities 



0% 



3 
5 

7 



Not 

Somewhat Serious Doesn't Apply 
Serious Serious At All or No Response 



13% 



8 
8 
6 



10 



10 



25% 

21 

14 

12 

24 

27 

25 

16 



18 



22 

28 
71 

10 

7 

36 



43% 

63 

62 

67 

43 

46 

51 

1 

66 



53 

50 

17-, 

64 
80 

35 



19% 

13 

22 

19 . 

21 

13 

10 

13 



12 



14 



13 



8 
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15a. (Continued^ 



Not 

Critically Somewhat Serious Doesn't Apply 

Serious Serious Serious At All or No Response 

Conflicting commitments 
on the part of project 

staff 3% 18% • 36% 355S 8% 

.Budgetary projilems 4 8 19 62 6 
Securing matching 

funds 3 6 11 66 ■ 14 

15b. Are there other difficulties you have encountered in project imple- ' 
mentation which have not described above? 

% of % of 

Total Respondents ^ 

18 27 No other difficulties or not applicable 

.7 * 11 Lack of cooperation from faculty or some 

faculty 

6 9 ' Changes in project staff 

4 6 Delays in construction/renovation 

4 6 Lack, of coc^)eration from institution 

administration 

4 6 Red tape in goihg through state, county, 

consortium channels 



16. What is the most serious difficulty your CAUSE project has 
encountered sad hoiir yas it handled? 

^ « of . « Of • 
Total Respondents ^ " 

21 22 ' Delays; too little timet missed deadlines 

13- 13 Too little productivity, inter-personal con- 

flicts, confusion over roles among project staff 

13 13 Getting cooperation of affected faculty 

12 12 Referred to an earlier question as containing 

the answer to this one 
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16. (Continued) 

% of % of 
Total Respondents 

8 9 Lack of cooperation by administration 

8 9 Inadequate budget > 

■ 6 7* Personnel changes 

4 4 Acquisition of equipment/software/materials 

4 4 None 

- ' 3 3 Informing students of services 

17. Areas of expertise that would have been .helpful . 

27% Evaluation . 

23 Con^uter applications 

15 Project management 

13 - Budget management 

13 Instructional development 

11 Other ^ 

8 Audiovisual media 

8 Equipment ordering 

8 Curriculum development 

7 Science/social science content experts 



18a. Are there incentives provided by your institution for working 
on CAUSE? 

44% Yes 55% No U No Response 

18b » Are ther& incentives provided by your institution for working on 
the CAUSE project? If '*yes"\ describe these incentives * 

« Of ' « Of ^ 
Total Respondents 

22 46 J^elease time for faculty working on project 

16 33 Verbal encouragement; campus bommunity 

recognition 

7 15 Summer support/ sup port for project-related 

- . activities 
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' 18b. (Continued) 

% of % of 
Total Respondents 

5 11 Stipend/merit raises 

4" 9 Don't know or not applicable 

3 7 Counts toward promotion/tenure 

18c » Are there incentives provided by your institution for working on i;he 
' CAUSE project? If "no" , what Incentives would have been helpful for 



^ achieving the project goals? 


% Of 

Total 


% of 
Respondents 




17 


31 


Release time 


12 


22 


Administrative recognition/encouragenrent 


11 


20 ■ 


Financial, incentives 

■ 


7 


14 


Don^t need Incentives 


6 


12 .... 


Promotion/tenure 


6 


12 


^ Item not applicable 


4 


8 


Institutional help in bringing about 
smooth implementation 


3 


6 


Additional staff J- 


HowQtnuch faculty release time or replacement time has been covered 


full or 


part time bg CAVSS monies? * 


% Of 
Total 


% Of 
Respondents 




64 


64 


Release time in academic year 


24 


24 


No i)^1ease time 


15 


15 


Summer stipends " 
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20a* In your opinion hov have faculty roemfaers at your institution 
changed as a result of the CAUSE pihjBct? Please describe ^ 
briefly as to: What do you expect the chMge to he ? 

S£ Of S£ Of : 
Jotal Respondents 

23 27 Faculty will learn hovt to develop curriculum. 

Individualize Instruction, develop materials 

23 27 , Faculty will develop mor€t expertise and positive 

attitudes toward using/teaching, cornputer 

19 22 Faculty will Improve teaching and Interact With 

students more effectively 

13 15 Faculty will learn content areas outside their 

own area of expertise . 

12 13 Faculty w^l 11 engage In new experience (research, 

planning, proposal writing, teaching off campus) 
and win have an enhanced sense of profMslonallsm 

7 9 Faculty will cooperate better Interdepartmentally 

6 7 Faculty will benefit from Improved facilities/ 

equlpnent ' ' v 

20b, In your opinion* hw have faculty m&wbers at your institution changed 
as a result of the CAUSE project? Please <iescr±be briefly as to f 
What impact has already been f&lt ? 

, S£ Of S£ Of 
Total Respondents 

21' 22 Faculty capabilities have, l?een broadened and 

" increased an<l faculty are engaging In new 
\ activities ^ : 

20 21 Teaching effectiveness has been enhanced and 

faculty have a better relationship with students 

16 17 \ Faculty have undertaken curriculum development 

-projects 

16' 17 Feculty have Increased use of computer and com- j) 

. puter-related materials ar)d/or Improved atti- ' 
tudes toward computing ■ - ^ 

14 14 Faculty In target departments have responded/ 

shSWn Interest 

6 . 7 Faculty morale Is Improved 

5 6 Faculty are benefiting from Improved ^ 

^ equipment/facilities 



. / 

/ 
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21 a » Jn your opinion^ hp^ have, the science/ cur riculvim and/or soms courses 
at gour institution changed as a result of the CAUSE project? please 
■ describe briefly as tor" What do you expect the change to be ? 

% Of % Of ' 

Total Respondents 



27 


28 


Add1t1orvof special components/features to 
specific existing courses 




17 


Modifications or innovation in existing courses 


16 ' 


16 


Better options/services, opportunities for 
students 


1 12 


12 


Development of new courses or^kinds of courses 


7 


7 ■ 


Improved instruction/faculty attitudes 


6 


6 


Proposed activities are being implemenrted 


5 


5 


Upgraded equipmegt/facilities implies 

improved curriculum ^ 




4 


Higher student' enrollment/interest 


4 




New/ revised materials 


In your 


^ < 1 

opinion, how have the science curriculmn and/or some courses 


at your 


institution changed as a result of the CAUSB project?'^ Please 


describe briefly as 


tot What impact has already been felt? 


Of 
Total 


Respondents 




19 


22 


Better options, services, opportunities for 
students 




21 


Addition of new c<Mnponents/features to 
existing courses 


M 


16 


DevelofMnent of new co^urses/sequences or 
kinds of courses 


, 8 


10 


Proposed activities are being implemented 


8 
8 


10 , 
10 , 


Top early oir no observed impact or NA (no 
' answer) or expectations not met 

Higher student enrollment/interest 
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21b. (Continued) 



X of % of 

Total Respondents 

6 7 Improved Instruction/faculty attitudes 

5 Modtflcatlons or innovation in existing courses 

5 6 New activities for faculty 

S ^ - . New/ revised materials 

4 5 Upgraded equipment/facilities Implies Improved 

curriculum ^ 1 



22a » In tjour opinion^ hoy have science Equipment And/or facilities at gour 
institution change as ^result '^f the CAUSE project and have the 
changes had the effects you anticipated? Please explain as tot what 
do you fixpect the change to be? 



/ 



% of 
Total 


% of 
Respondents 




22 


. 26 


N^equipment, materials/facilities 


20 


23 


Enable, bring about, instructional improvement 


16 


18 


Upgrade, expand, improve equipment/facilities 


13- 


15 


Obtain new computer equipment 


10 


11 


No Impact/not applicable 


8 


10 


Upgrade> expand^ Improve computer equipment 


2 


2 


Improved faculty and/or student morale 


2 • 


2 


Positive Impact outside CAUSE project 
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22b » In your opinion hoy have science equipment and/or facilities at your 

institution changed as a result of the CAUSE project? Please describe 
briefly as to: What impact has already been felt ? 



* of 
Total 


% of 
Respondents 




23 ■ 


26 , 


Enable, bring about improved instruction 


14 


16 


* 

New equipment, materials, facilities 


11 


12 , 


No impact/not applicable 


10 


11 . 


New computer hardware/ software 


10 


■11 ■ 


Upgrade* expand, improve equipment/facilities 


8 


10 


Proposed activities have beea implemented 


- 7 


8 


Positi-ve impact outside CAUSE project 


6 


7 


Improved faculty and/or student morale ^ 


5 


6 


Upgrade, e)ipan;i, improve computer equipment 


Xn your opinion bow b&ve students at your institution changed as a 


resuJt of the CAVSS project? Pleasie explsdn as to: Whai do 6ou 


expect the chanSP to be? " 


% of 
Total 


% of * 
Respondents 




'5,0 , 


55 


Improved training* student competency 


20 


. 22 


Improved student;^ attitudes toward target 
courses and the delated disciplines [ 


t 

11 


/ 

12 


Higher student enrollments 


10 


11 


Students are mor^e proactive* participate more 


3 

/ 


, 4 


* * 

Too earty to tell or, not applicable or don't . 
ktiow 



/ 
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23b.' Xn your opinion hm have students at tjour institution changed as a ' 
Tmault of tt» CAUSE project?^ Pieaso explain as to: What impact has 
ilroady bean felt ? '> r-^^ 

i of of 

Total Respindents • ' 

30 ^ 32 ' Improved training, student competency 

21 23 Students are more pt^oactlve, participate more 

16 17 NA or too early to fell or don't know oji lesfe^ 

impact than expected * *^ 

14 15 Improved student attitudes toward target courses 

10 10 Implementation of activities is proceeding/ ^ 

^tudents are changing or have changed 

.,8 9 Higher student enrollments 



24 * Have an^ unexpected changss occurred a result of the CAUSE ptoject? 
Please describ^ them * 

% of % Of , / 

Total Respondents ^ : 

20 24 ' None * ^ ^ . 

16 19 Unexpected outcomes; spin-offs, from CAUSE 

11 13 Higher *ficulty us«/acceptance, than expected 

8 ^ 10 Higher student use/acceptance than expected ' 

7 / 9 General impact and/or impact on the conmunity 

greater than expected 
■* ' 

6 8 Faculty more involved in new activities than 

expected , 

4 '5 More negative impacts than expected 

3 ^4 'Better faculty morale and collaboration than^ 

expected 



53 



25* If, you had the chance to start your CAUSE project over agaiiif from 
. the beginning, wMt Mfould gou do differently ? 

St Of St Of ' 

Total Respondent ^ - 

22 23 Better front-end planning 

■ 17 17 No changes' 

16 - 16 More release time and/or sunnier support for 

faculty and project staff 

15 15 Increase budget ' ^ 

14 14 Make changes in staffing 

12 12 Changes in strategies, activities, equif^nent 

10 10 More faculty participation/cooperation 

6 7 Better managern^nt^ 

4 4 Change evaluation plan 

2iSa* Have you sought or will you seek funding to continue CAUSE activities? 
705E Yes 215E No 105E No response 

26b* Have gou sought or will jjoii seek funding from other sources to continue 
activities started under CAUSE? If ^tfes"^ from what sources will 
you seek funding ? ^ ^ n 



% of 
Total 


% of 
Iftespondents 




30 ■ 


39 


NSF 


28 


38 


Private foundations, individuals, corporations 


20 


26 


OwnMnstitution 


16 


21 


Don't know 


12, 


15 ' 


Federal agencies other than KSF 


4 


6 


State agencies 
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26c, ^ Have you sought or vill you seek ftmding from other sources to con- 
tinue ttotivitles started under CAUSE? If "no" yiJJ activities 
started as a part of the CAUSE project, cemtinue after CAUSE funding 
has ended? 



% of ^ 
Total 

38 



% of 
Respondents 

IDO 



Yes - can operate within coll ege/uni vers itij/ 
consortium budget 



27, Evaluation can best be described as the following : 



Strongly S.trongly No 

Agree Agree Disagree Disagree Response 



Evaluation is important to the 
institution in monitoring the 
effectiveness of projects of- C 
,this type ^ 39$ 

The best way to^valuate a 

,CAUSE project is to have an - 

expert(s) from outsidft.our - , 

institution review bur 

project outcomes ^ ^ 12 

The primary reason evaluation .*> ^"""^ 
is included in our project is . , 
mostly to ineA CAUSE require- ^ 
ments for evaluation 7 

A clear ind yiorough descrip- 
tion of our project will meet 
CAUSE requirements for project 
evaluation 4 

Evaluation results have been 

used to change some of the 

activities and/or outcofnes 

of this project 20 

Our project funds allocated 

for evaluation activities . 

Could be better spent on 

other project activities 5 

Fonnal evaluation activities 

take up too much time, effort, 

and money on our CAUSE project 6 



54% 



21 



31 53 



28 J3 
t 



43 36 



48 20 



22 53 



56 



1$ 



15 



13 



14 



13 



1$' 



10 
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Negative evaluation results on 
our CAUSE^project might jeopar- 
dize our institution's chance 
at further external funding 
for science instruction 

Project evaluation means cori- ^ 
ducting activities which have 
little or no usefulness ta.our 
CAUSE' project staff 

Project evaluation is best 
accomplished and most highly 
useful when it is Conducted 
internally by proje.ct personnel 
in an ongoing manner 

The best way to conduct 
evaluation of our CAUSE project 
is to try to measure student 
achievement gains 

Given the nature of our project, 
evaluation is really an 
irrelevant activity 

, Evaluation activities have not 
gtfne as planned 

To do the kind of evaluation 
we would like, We need more 
money and staff ■ , 

We are not .qualified to do an 
.internal evaluation of our 
project 



Strongly Strongly No 

Agree Agree Disagree Disagree Response 



18 



10 



16 57 



12 67 



54 22- 



37 44 



55 



34 53 



27 



54 



12 



20 



34 

7 



14 



12 63 y- 



22 



56 



If you hAd the opportunity to suggest strategies for promoting project 
SMCcess to a prospective CAUSE project director, what would be your ■ 
recotanendation? 



% Of 

Total 

24 



21 
18 
14 

14 

7 

6 
5 
3 



. % of 
Re'sportderjts 

26 



23 
2D 
15 

15 
12 
8 

7 
6 
3 



Make sure that management and authority Issues 
are carefully wprked out 

Do careful front-end planning 

Get the participation/commitment of the faculty 

Get the cooperation/commitment of the 
administration 

Implement carefully with attention to detail 

Select staf# carefully s 

Get release time for faculty -and for staff 
development 

Make sure communication is frequent and adequate 
8e willing to dedicate yourself ' . 
Use evaluation formatively 



If you Aad the opporttmitg to advise a prospective CAVSB project ^ 
director about CAVSE project evalvatidni what would he your primary 
reco^s^xndation? 



% of 
Total 



% of 
Respondents 



22 


24 


use outside experts 


2D 


22 


Know what you want; be realistic;'^ satis- 
fied with plan 

* { 

Do formative evaluation 


13 


14 ' 


13 


14 


use internal people for evaluation 


U 

i 


11 

< 

7 


Do objectives-based evaluation 
Start^early 


3 


3 


Avoid traditional evaluation technique^ ' , 
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30* ' Xf you h^d the opportuhitg to make' one , suggest ion to^the 'nsf<:avsB 
progr^ office to improye the CAUSS pxogx&mt >fhat recoifsnendation 
wdijld gou make? 

St of , of ' ■ - 

Total ' Respondents , , ' ' 

' ' ' . ' y 

40 , 43 , More/better comnunlcdtiort, disseminati^fn among 

Ijrojects and with. NSF staff a 

21 23 / None - doing a good job . * . , 



7 8 ' ' More-flexibility for project directors; leSs 

red tape ^ - 

5 6 * Hore.fundihg/foVlow-up/changes ' . / . 

5 6 Better evaluation policy ■ 

4 5 Make changes^ io policies arQund awards 

4 ' 5 institutional conmitment should be assured . 

3 3 . ^viewers should be in fields of subject of* ^ 

proposal or from same kind of institution ' 

3 3 institutions shouldnU be required to change/ 

modify to please NSF 
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CONTENT ANALYSIS OF FUNDED PROPOSALS* 1976-1979 



Purpose of the Content Analysis 
A- content analysis of funded prpposals was chosen as a broad focus 

t ■ • t 

eva-luation activity because of Its^ advantage in supplying data from 

. ■ * * " ■ 

available documents. As -a result, there was no need to gather swne kinds 
X)f information frocn'lndividu^l sites. It also offers "the opportunity to 
st^dy the terior'/of CAUSE projects prior to their implementation. These 
.data can'then be viewed In relationship. to the other data collection 



activities* the surveys* site visits* and case studies. 

The content analysis provides 4i|fon^ relevant to certain aspec^ts 
df the. evaluation' Issues of concern. For example* proposals contain 
specification of Institutional needs* Institutional goals and objectives* 
and methodology of project ImpTementatlon- Through the analysis of pro* 
posals* these areas were categorized and compared across such variables 
as project year and type of Institution. Artother function Of the content 
analysis was to look at trends In science education as described by pro- 
posing Institutions. Therefore* the primary purpose^^f the content 
+ 

analysis of funded proposals was to provide baseline data to support and 
drive other evaluation activities ► . 

CAUSE proposals provide a valuable source of information about per- 
ceived needs and goals and the planned strategies for meeting thi needs 
and achieving the goals. They are statements of what could be, and 
further what ought to be» in the eyes of proposers. 
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Content Analysis for this Evaluation ^ 

Category se1ect1or\ . The evaluation team at DEA made changes in categories 
earlier treated by NSF's Office of Program Integration iri order to make the con- 
^nt analysis data more useful in the scheme of the total evaluation. New date- 
gories and sub-categories were added while others were expanded to add detail or 
were collapsed for clarity. The development of a final set -of categories'has 
been the result of a ,long and careful team effort Xp produce categopi€t that are 
not too ambiguous or overlapping, and which arise most naturally from data in the 
proposals* The major variables used in the content analysis are; 

1. Institutional type 

2. Discipline 

3. Audience 

4. Problems and needs 

5. Goals and objectives 

6. Outcomes 

.Each pf these has been divided into categories and subcategories » All the catfe- 
gories aijd subcategories were chosen with the intent of maximizing inter-reader 
* reliability in data collection* 

Institution type and discipline . The first two variables, "institution 
type" and "discipline" are self-explianatory: institution types are exactly the 
same, as those described, by NSF two*year, four-year, Ph.D granting and consor- 
tium. Discipline includes the rriijor disciplinary areas defined by NSF for use by 
the proposers. ^ * 

Audience . The "audience" variable contains six categories which differ 
~~ — f 

somewhat from those identified by NSF» This variable defines that groupr (or 
groups) for which the proposed project is meant* The six categories under 
audience are: 
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1* Faculty 

2» Local conmuTiity ^ 

3, Majors and Non-Majors; Introductory ^ 

4» Science Majors; Introductory ^ 

5» Science Majors; Advanced 

6» Non-Science Majors 
Problems and needs - NSF divided "problems" and "needs" Into two categories 
where "needs" described the kind of interventio^i that has been identified as most 
likely to solve institutional problems and where "problems" meant an identifiable 
lack, an area requiring action. In our classification system, "problems and needs** 
have been collapsed into one .variable which has the same focus as the "problems" 
section for NSF; that is, an identifiable lack. The categories under "problems 
and needs" which we identified are: ' \ 

K Curricyla need revision/addition to keep pace with 
cucfent state of science education. ' * 

2. ^ Teaching methods are nGt as effective or efficient 

as they shot/ld be. 

3. Faculty need to update knowledge or skills. 

4. Missing/inadequate hardware, software, facilities. 

5. Student problems necessitating curricular or 
instructional revisions. 

It win be noted that this classification system is very similar to 

. "problems" categories except that the student problems category is ^ 
broader, allbwing for greater inclusiveness. and there is the addition of 
the "missing/inadequate hardware, software, and facilities", a category' 
that is covered in NSF's "needs" section. .. - — . 

Goals and objectives . The next major variable devised by UEA is 
called "goals and objectives". It is roughly equivalent to NSF's "needs" 
, in that it identifies the desired approach to solving the problem or 
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satisfying the need. It identifi'^, in other words, the purpose of the 
Innovation. The categories are; 




K To acconinodate students at their levels and^ 
for their needs* 

2. To update' curricula to keep pace with the current 
state of science education* 

3. To improve teaching methods to make them more efficient 
or effective* 

4. To provide for faculty development. 

It will be noted that where NSF identified seven major desired approaches 
to solving problemstwe have identified only four. We originally left 
out NSF's four categories which were related to equipmentt materials* and 
laboratories and included them in our "problems and needs" section. - How- 
evert as we analyzed the 1976-197^ group of proposals, we ended up writing 
goals and objectives related to equipment/facilities use and acquisition 
in the "other" column* As a result* when revising the content analysi? 
categories for the 1979-analysiSt we added a fifth category to^"go^ls 
and objectives": equipment and facilities acquisition. Data in the tables 

reflect this addition. 
* 

Outcomes . Our final major variable is "outcomes"* "By ^'outcome" 
vJ>^mean the strategy or strategies chosen by an institutior^ to meet its 
goals and objectives* To put it another way* the variable identifies* 
specifically, what the grant will be used to do* The categories Are; 

1. Faculty development 

2. Remediation/individgalized^Hnstruction 

3. Curriculum addition/revision \^ 

4. Use of computer^ ' 

5. Equipment/mate/ials/facilities \ 
' \ ■ ^ 

These categories are roughly equivalent to those o^^NSF's "strategies". 

. • • \' 
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During the first stage of the content analysis, we collected and 

analyzed data from funded proposals from funding years 1976-1978, Sub- ^ 

sequently, we analyzed 1979 proposals. Using what we had learned during ' 

the first analysis, we eliminated a few empty categories arid added 

"equipment and facilities acquisition" to our "goals and objectives" 

category* In developing the original set oT categories, it was assun^d 

that equipment acquisition would not stand alone 4S a goal. However^ in y 

reviewing the/^(rst round of proposals, we often found institutions siting 

acquisition as a goal. It was also determined that equipment acquisition. 

often consumed a significant portion of the budget. Hence, the category 

was added for the 1975 review. Other smaller subcategories were added^o 

the components of the outcotnes. AH cross-t^bluajions were then recomputed 

and include all funded proposals from the four years, 1976-1979. 

V\e general project variables . Table. 4 depicts major categories 

and sub-ca.tegories of problems and needs reflected in the 273 funded 

proposals. The most frequently cited categories were: 

Hardware and software are missing and/or laboratory or learning 
center facilities are inadequate (515C); 

txistance of student problems which necessitate curricular or 
instructional revisions (415C); 

The curriculum is in nee<t of additions or revisions (40*)* 

*-\ ■ - ' 

Again, we assumed that projects\iight reflect mora than one problem or 

■ \ . ' / 

need* Therefore, tabled figures reflect a dupVidated cou/it* 

Table 5 presents the goals and\, objectives of th§ funded CAUSE pro- 
jects. As previously explained, DEA dild noH^originally create a category 
for equipment/facilities developtnent in the goals and objectives section,^ 
but added that category iji the 1979 analysiV Therefore,, the "other" 
category (45S) includes equipment and/or <fac\nties from the 1976^ ^ 
1978 proposals. The shift in this area wili b\ more clearly seen when 
the data are analyzed over project initiation yaars. Of, the remaining 
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categories, two were most often cited: \ 



V. 



To update curricula In order to keePlwce with the current 
state of science education (46X) 

T^mprove teaching methods In order to make them more 
efficient and effective (39%) 

As^ before, we allowed 4 maximum of three categories of goals and objectives 

to be cited by each Individual project. NTherefore, figures reflect a 

duplicated count. * y 

.The major outcomes and primary outcome" of all funded CAUSE projects 

are shown In Tables 10 and 11. In the first tsble, figures reflect couats 

of a maximum of three major outcomes while the second shows a primary out- 

come for each of the 273 funded projects. The three most frequently cited 

major outcomes were: 

Equipment, materl^s, and/or fadllltles (61%) 
^ Computer" acquisition and/or applications (51%) 

Curriculum' additions or revisions (49%) ^ 
However, the figures change somewhat when only primary outcomes are 
analyzed: the most frequent^ly cited categories remain the same and occur 
in relatively s^inilar proportions, biit faculty develofxnent projects fall 
from 231 to This factor indicates\he secondary nature of faculty 
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development among the CAUSE prpjects. . \ 

In reviewing the consistency of response among this 

^ \ 

tables, some Interesting patterns emerge. For example, need for faculty 
skill and knowledge development was reflected \n 40 of the proposals, while 




faculty development as a goal, and 63 mention it as 
IS to^dicate that while faculty development is not 
jp(n)blem, it is often thought^of as a sol-ution to 
often student needs).. It may also be the 
t would occur as a result of^anges in 
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teaching methods or equipment/facilities acquisition or adaptation. ^ More 
projects cited curriculum additions or revisions as an outcore (n=134) than 
as a need (n=109). Again, this IntHcates the utilization of currlcular 
change to solve other types of problems. In tabulating responses, we 
also found many institutions citing student problems but suggesting com- 
puter acqjulsitlon or equipment and materials acquisition as objectives and 
outcomes. This again reflects the pattern of using a variety of means to 
solve student problems. 

,/ 
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■ Table 4 

Problems and Science Education Needs 
Frequencies and Percentages 



«6 



Problem or Need 



All Funded Proposals 



Curriculum needs revision or 
additions due to: 

V Inadequate coverage 

Changing goals 

^ Other reasons 

^Mchlng methods ar€ not as efficient 
or effective as they should be 

Faculty need ta update knowledge or 
skills In the following areas: > 

Instructional techniques 

Subject matter 

Computer skills^ 

* 

Hardware and software are missing 
and/or facilities are Inadequate 

student problems which necessitate 
currlcular or Instructional 
revisions due to: 

Inadequate preparation o^ 
Incoming students 

Poorly motivated students 

Poor success rate of students ■ 

Increasing diversity of the 
student population 

^ther problems or needs 



109 
75 
20 
17 




40 

9 
6 
28 

140 

HI 

53 

7 
25 

W * 

10 . 



40 

V 
7 
6 

22 

15 

3 
2 

10 
51 

41 

19 
3 
9 

20 
3 



Total 



273 



^ 100% 
'■ 



— ' ■■ — e 

^Proposals may address. more t^an one problefO or need* Therefore, frequencies 
and percentages reflept a duplicated count* In the content, analysis, the 
number of problems or needs which- a proposal could be listed as addressing 
was limited to three* This did not eliminate a significant number of prob- 
lems because very few 'proposals d1scusse<t more than three* 



1 



67 



Table 5 



/ 



Goals and Objectives of CAUSE Projects 
Frequencies and Percentage? 



An Funded Proposals 



Goals and Objectives 
/ 



To acconmodate students at their levels 
and/or for their needs 

' To update curricuU in order to keep pace 
with the current state of science education 

To improve teaching methods in order to make 
them more efficient and effective ~ 

To providfe for faculty development 

,■ ■ * 
Equipmient and facilities acquisition 
(data on '79 only, n»72) 

Other (includes equipment and facilities 
, from 1976-1978) j 



61 
125 

107 

61 

52 
123 



22 

46 

39 
22 

72 

4& 



Total 



275 



100% 



Proposials may address mor^ than one goal or objective. Therefore, frequencies 
^nd percentages reflect a duplicated c;uwt. In the content analysis, the 
number of goals anjl^ objectives which a proposal could be listed as addressing 
was limited 'to three.- This did not eliminate a significant number'of goals 
and objectives because very few proposals discussed more than three. 
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Table 6 



Outcomes of CAUSE Projects 
Frequencies and Percentages 

/ 



Outcome 




-7" 



acuity Development 



/ 



Individualized Ins^uction/Remediation 



Curriculum Addition/Revision 



Computer Acquisition/Applications 



Equipment/Materials/Facilities 



All Funded Proposals 
f % 



./ 62 
134 

138 

J 

167 



23 
22 
49 
51 
61 



Total 



273 



100% 



Proposals often describe more than one outcome. Therefore, frequencies 'and 
percents reflect a. duplicated count. In the content analysis, the number 
of outcomes which a proposal could be listed as addressing was limited to 
three. This did not eliminate a significant number of outcomes because very 
few proposals described more than three. 
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Table 7 

1 Primary Outcofne'of Each CAUSE Project 
Frequencies and Percentages 
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All Funcled ProposaU 



Primary Outcome*^ 



Faculty Development 


8 




3 


Individualized ^ ^ / 

Instruction/Remediation ' / 

/ 


■ 43 




16 


Curriculum Addition/Revision / 

/ 


77 




'28 


Computer Acquisition/Applicatign 


67 




24 


Equipment/Materials/Facilities 


75 




27 



Total 



273 



1005! t^. 



^One primary outc^Sme was listed fdr each project. 
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